Tape library emulation with automatic configuration and data retention

ABSTRACT

Disk based emulation of tape libraries is provided with features that allow easier management and administration of a backup system and also allow increased flexibility to both archive data on tape at a remote location and also have fast restore access to archived data files. Features include automatic emulation of physical libraries, and the retention and write protection of virtual tapes that correspond to exported physical tapes.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims a priority benefit fromU.S. application Ser. No. 13/613,373 filed on Sep. 13, 2012 and entitledTape Library Emulation with Automatic Configuration and Data Retention,which is a continuation of and claims a priority benefit from U.S.application Ser. No. 11/356,726, filed on Feb. 17, 2006 and entitledTape Library Emulation with Automatic Configuration and Data Retention,which claims priority to under 35 U.S.C. Section 119(e) to ProvisionalApplication 60/654,714, filed on Feb. 17, 2005. The disclosures of theseapplications are hereby incorporated by reference in their entireties.Furthermore, any and all priority claims identified in the ApplicationData Sheet, or any correction thereto, are hereby incorporated byreference under 37 C.F.R. §1.57.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to systems and methods for storing electronicdata and has applicability to enterprise data backup systems.

2. Description of the Related Art

Improving backup and restore performance is a continuing desire ofenterprise data managers. In a typical computing environment, magneticdisk drives are used as the primary storage mechanism for active data,whereas magnetic tapes are used for data backup and archive. Themagnetic disks provide rapid and reliable access to data, but they areperceived as being more expensive. In addition, since they arenon-removable, they are at risk of physical disasters. Magnetic tapestorage is perceived as being less expensive and, because tapecartridges are removable, they can be moved to offsite locations toprotect against physical disasters. Therefore, most backup software inuse has been optimized for use with magnetic tape technology.

Reading and writing data on a tape requires that the reel be unwounduntil the desired location is found. Once in the appropriate location,the read or write operation can begin. Because of the mechanical natureof this access, read and write operations are slow and often fail. Inmany situations, it would be beneficial to provide the random accessspeed and the reliability of a magnetic disk drive to backup systemswhile still allowing for the possibility of offsite storage. As aresult, a new category of magnetic disk systems is becoming popularcalled virtual tape technology.

Virtual tape systems are magnetic disk systems that transparentlyemulate a tape drive and/or a tape library. They provide the samephysical connections to a host, such as SCSI, Fibre Channel or Ethernet.This allows them to connect in the same way as the tape systems they arereplacing or augmenting. They also provide the same logical response totape drive and robot commands, which allows the same backup software toremain in use. The emulator is also able to send the host computer theexpected tape-drive interrupt signals such as beginning-of-tape,end-of-tape, and inter-record-gap. In this case, such a system can plugright in to an existing tape based storage system without a need for theuser to change the storage network or software environment.

Although such systems have been successful in the marketplace, thecurrently available devices still do not fully take advantage of theproperties of disk storage in a way that provides maximum flexibilityand usefulness.

SUMMARY OF THE INVENTION

In one embodiment, the invention comprises a method of emulating a tapelibrary data storage system using one or more hard disk drives. Themethod comprises querying one or more physical tape libraries in a datastorage system to acquire a configuration of the one or more physicaltape libraries in the data storage system. Data storage space isallocated on the one or more hard disk drives to virtual devices of oneor more virtual tape libraries, wherein the one or more virtual tapelibraries comprise virtual devices emulating physical devices in theacquired configuration of the one or more physical tape libraries in thetape storage system. Data storage space is also allocated on the one ormore hard disk drives to at least one additional virtual deviceassociated with the one or more virtual tape libraries, wherein theextra virtual device has no corresponding physical device in the datastorage system.

In another embodiment, a method of emulating data storage on a magnetictape media using one or more hard disk drives comprises allocating datastorage space of the one or more hard disk drives to one or more virtualtape libraries, wherein the one or more virtual tape libraries compriseone or more virtual devices emulating states of one or more physicaldevices of the one or more physical tape libraries in a tape storagesystem, and storing data in the storage space allocated to the one ormore virtual tape libraries according to a first user defined periodicschedule. This method further includes replicating the data stored onthe one or more virtual tape libraries onto the one or more physicaltape libraries according to a second user defined periodic schedule.

In another embodiment, a method of handling data storage on a hard diskstorage system implemented to emulate one or more attached tapelibraries comprises requesting export of at least one physical tape froma physical tape library and write-protecting the data on the hard diskstorage system that is associated with the virtual tape corresponding tothe physical tape to be exported.

In another embodiment, a method of emulating a tape library data storagesystem using one or more hard disk drives comprises allocating datastorage space on the one or more hard disk drives to virtual devices ofone or more virtual tape libraries, wherein the one or more virtual tapelibraries comprise virtual devices emulating physical devices of one ormore physical tape libraries in the tape storage system. The methodfurther includes allocating data storage space on the one or more harddisk drives to at least one additional virtual tape library that has nocorresponding physical tape library in the data storage system.

In another embodiment, the invention comprises a storage systemcomprising at least one disk based storage appliance and at least onetape library. The disk based storage appliance is configured to respondto commands generated by backup software as at least one emulated tapelibrary. The disk based storage appliance stores data files in anemulated tape library accessible to the backup software that are alsostored on tapes that have been previously removed from the tape library.Thus, tape archive and disk based read access to a set of data files issimultaneously provided.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic of one embodiment of a data backup system in whichthe invention may advantageously be used.

FIG. 2 is a functional block diagram of certain components of anembodiment of the backup system of FIG. 1.

FIG. 3 is a flow chart of the operation of one embodiment of the systemof FIG. 2.

FIG. 4 is a flow chart of a method of configuring a virtual tape libraryin one embodiment of the invention.

FIG. 5 is a functional block diagram of a virtual and physical libraryconfiguration in one embodiment of the invention.

FIG. 6 is a flow chart illustrating a method of physical and virtuallibrary synchronization.

FIG. 7 is a functional block diagram of a virtual and physical libraryconfiguration in another embodiment of the invention.

FIG. 8 is a flow chart of a method of disk storage data retention in oneembodiment of the invention.

FIGS. 9A-9C are functional block diagrams of virtual and physicallibrary configurations in another embodiment of the invention.

FIG. 10 is a flow chart of the operation of a virtual shelf in thesystem of FIG. 2.

FIGS. 11A and 11B are functional block diagrams of virtual and physicallibrary configurations in another embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Preferred embodiments of the present invention will now be describedwith reference to the accompanying Figures, wherein like numerals referto like elements throughout. The terminology used in the descriptionpresented herein is intended to be interpreted in its broadestreasonable manner, even though it is being utilized in conjunction witha detailed description of certain specific preferred embodiments of thepresent invention. This is further emphasized below with respect to someparticular terms used herein. Any terminology intended to be interpretedby the reader in any restricted manner will be overtly and specificallydefined as such in this specification.

FIG. 1 illustrates one example of a system including a hard disk basedappliance with tape emulation features that can be used in a dataprotection environment. In this system, application servers 12 areconnected to each other and to a backup server 16 over a network 14. Inone embodiment, the backup server 16 communicates directly with the diskappliance 18 and has no direct communication to the tape system 20. Inthis embodiment; the tape system 20 is under the control of the diskappliance 18 via a SCSI, iSCSI, Ethernet, Fibre Channel, or otherprotocol communication link 21. It will be appreciated that multipletape systems may be connected to communication link 21.

Backups from application servers 12 are received by appliance 18 (viathe backup server 16) and are written to disk based (preferably RAID)storage of appliance 18. The disk appliance 18 may include an internaldisk drive array, and may alternatively or additionally connect to anexternal disk drive array through a storage adapter which may, forexample, be configured as a Fibre Channel or SCSI interface.

Appliance 18 may then automate the process of transferring the datastored on disk media to physical tape media in tape system 20 forarchival purposes. As explained further below, the transfer of the diskstored data to physical tape media may be done without user interventionon a periodic basis. Furthermore, the appliance 18 may periodicallymonitor the tape system 20 for changes such as tape import or export (atape being installed or removed from one of the physical tape libraries)and generate appropriate actions to ensure that the RAID storage virtualmedia emulates the physical media on tape system 20.

It will be appreciated that the hardware components, functionality, andsoftware present in the backup server 16, disk appliance 18, and tapedrive/library can be combined and/or separated in various ways. Forexample, the disks of appliance 18 can be located in a separate device.As another example, the tape drive/library 20 hardware and functions canbe integral with the disk appliance 18 rather than provided as aseparate unit. As described above, the appliance 18 can be configured tointeract with the backup server 16 in exactly the same manner and formatof communication as the tape drive/library 20. In this way, software onthe backup server 16 that is configured to communicate and store datausing tape commands and tape data formats can utilize the disk basedappliance 18 without modification. Speed is still improved in many casessuch as restore operations, however, because tape commands such asmoving to a desired block can be accomplished on disk with the virtualtape much faster than a physical tape cartridge in a physical tapedrive.

FIG. 2 is a functional block diagram of certain components of anembodiment of the backup system of FIG. 1. In the example of FIG. 2, thedisk appliance 18 has a first communication link 23 (of any protocol)connected to a backup server 16. The backup server will typicallycontain a backup software program 24A that controls data transfer fromthe application servers 12 (FIG. 1) to the appliance 18. The diskappliance 18 will also typically host another software program 24B thatis used to configure the appliance 18, and define how the appliance 18responds to commands and data received over the link 23 from the serverbackup software 24A. Software 24B can wholly or partly reside in memoryin the appliance 18 and/or the backup server 16. In one embodiment, thedisk appliance software is accessed via a browser program on backupserver 16 or any other computer on the network.

The appliance 18 is also coupled to another communication link 21 thatis connected to three physical tape libraries (PTL) 20A, 20B and 20C.More or fewer tape libraries may be provided, it will be appreciatedthat three is merely an example. In some embodiments, a single physicaltape library can be partitioned to behave as if it were multipleseparate tape libraries, as described in U.S. Pat. No. 6,328,766, theentire disclosure of which is hereby incorporated by reference. It willbe appreciated that any or all communication links connecting thedevices of FIG. 1 could be over the same network link.

The disk appliance 18 in the example of FIG. 2 is configured to includethree virtual tape libraries (VTL) 22A, 22B and 22C. The VTL1 (22A)contains virtual devices emulating the physical devices of PTL1 (20A),while virtual devices in VTL2 (22B) and VTL3 (22C) emulate the physicaldevices of PTL2 (20B) and PTL3 (20C), respectively. The “emulation” oftape libraries in disk appliances is known in the art, and those ofskill in the art understand and can create hardware and softwarecomponents for an appliance 18 that can emulate tape libraries.

Generally, tape library emulation is understood to mean that theappliance 18 responds to commands and data transfers from the backupserver with responses and data that the backup server expects from atape library being emulated, even though no physical tape library is indirect communication with the backup server 16. In the discussion thatfollows, manipulation of both physical objects and virtual objects isdescribed. When such terminology is applied to virtual objects, such as“moving” or “creating” a virtual tape, or “allocating” some portion ofdisk appliance 18 to a virtual object, these terms are intended asgenerally used in the art to mean that the disk appliance 18 isconfigured or re-configured to respond to commands from the backupserver with the same responses that would be produced by a physicaldevice having characteristics corresponding to the virtual device in itslatest virtual configuration.

Although emulation of tape libraries in disk based appliances is known,their usefulness has been limited due to complexities in the managementof the combined disk and tape storage environment. In accordance withthe inventions described herein, data storage systems with improvedproperties, data access, and simplified management are provided.

It is often desirable for the emulated tape libraries 22 to beconfigured identically to the physical tape libraries 20. To accomplishthis in an efficient and easily administered manner, the appliance 18may be configured to use standard query commands to detect theconfiguration of the physical tape libraries such as the number and typeof tape drives, the number of storage slots, etc. Upon receiving thisinformation, the appliance 18 can be configured automatically to emulatea tape library identical to a given physical library coupled to theappliance 18. In addition, if a new PTL, PTL4 (20D) for example, isadded to the tape storage system, the disk appliance can query the PTL4(20D) and then emulate the discovered physical devices with virtualdevices in VTL4 (22D).

The disk appliance software 24B comprises user interface software tocontrol the configuration of features of disk appliance 18. Featurescontrollable by the disk appliance software 24B described further belowinclude, for example, control over replication of the data stored on theVTL's by the backup software 24A to the PTL's for synchronization, dataretention time limits upon removal of tapes from the PTL and virtualdevice configuration management software.

During operation, the backup software is configured to perform backupand restore operations to PTL1, PTL2, and PTL3 as defined and managed bythe system administrator as if the appliance 18 was not present. This istypically performed on a periodic schedule fixed by a systemadministrator via backup software 24A. The commands and interactions arereceived by the appliance 18, and the appliance 18 interacts with thebackup server 16 as if it were a collection of PTL's. As describedabove, the data stored on the appliance 18 by the backup software isperiodically or on command (via the appliance software 24B) transferredto the physical tapes in the physical tape libraries 20 so that the dataintended for storage on tapes in the PTL's is physically present onthose tapes within reasonable and desired time frames defined by asystem administrator. A second periodic schedule different from theperiodic backup schedule can be used for synchronizing data in thevirtual libraries with data in the physical libraries.

It is one aspect of some embodiments of the invention that the diskappliance can implement an additional emulated tape library, referred toherein as the “virtual shelf” (VTS) 26. The shelf 26 can be used toallow access to data on tape cartridges that have been exported from thePTL's. This feature is described further below.

FIG. 3 illustrates a process that may be performed by the system of FIG.2. Referring now to FIG. 3, process 100 starts at step 105 by performingconfiguration management of the various virtual devices in the diskappliance 18. FIGS. 4 and 5 discussed below provide additional detailconcerning this process that can be performed in some inventionembodiments. Disk appliance 18 can query PTL's connected to it viavarious network connections to acquire configuration information at step106. Configuration information acquired at step 106 can include newlyadded components such as the PTL4 (20D) shown in FIG. 2. Configurationinformation changes can also include modification of existing hardwareand deletion of existing hardware in the attached PTL's 20.

Process 100 continues at step 110 where data transferred under controlof backup software on the backup server 16 is received by the appliance18. As discussed above, the backup server 16 communicates data to bebacked up on PTL 20 directly to disk appliance 18. Disk appliance 18stores the received data to the various virtual devices that correspondto the physical devices requested by the backup server 16. Process 100continues at step 115 where data stored on VTL 22 is replicated on thePTL 20 so as to synchronize the data between the VTL 22 and the PTL 20.This replication may be periodic (e.g., every day, every 12 hours,etc.). The replication policies may be user settable as discussed above.

At step 120, the process 100 continues with the monitoring of the PTL(s)20 for detection of imported or exported physical tapes. The importand/or export handling acts are carried out at step 125. Several examplecases of tape import/export and the handling of these cases arediscussed below. It will be appreciated that data backups of step 110,replication to the physical libraries of step 115, and the import/exporthandling of tapes are performed periodically and/or on command asrequired in any order on any desired schedule in an ongoing mannerduring operation of the system.

Referring now to FIGS. 4 and 5, virtual library configuration managementin some embodiments of the invention is illustrated. In the example ofFIG. 5, the PTL 20 contains one physical tape drive 28, a robotic mediachanger 30, five tape storage slots 32 (labeled 1 through 5) containingfive tape cartridges labeled A-E, and a tape export slot 34.Accordingly, during step 106 of FIG. 4, the appliance 18 queries the PTL20 do detect the PTL configuration. After gathering the information, thecorresponding VTL 22 is created in the appliance at step 107, emulatingthe hardware of PTL 20 to contain one virtual tape drive 36, a virtualmedia changer robot 38, five virtual tape slots 40 (labeled V1 throughV5) containing five virtual tape cartridges labeled VA through VE, and atape export slot 42. The disk appliance 18 may implement the virtualmedia changer 38 as a standard SCSI media changer device, as defined inthe T10 SMC-2 document. Each virtual tape drive 36 inside the VTL 22 maycomprise an emulation of a standard SCSI sequential device, as definedin the T10 SSC-2 document. The process of “creating” or “implementing”the virtual library typically comprises the creation of data filesstored in the control/memory circuits 31 of the appliance 18 that definethe appliance response to commands and communications received from thebackup server 16. The data format, organization, and disk spaceallocation to implement virtual tape cartridges in the disk appliance 18may be performed as described in U.S. patent application Ser. Nos.11/215,740 and 10/943,779, the entire disclosures of which are herebyincorporated by reference in their entireties.

In addition to emulating physical devices of attached PTL's 20, diskappliance 18 can (at step 108) receive requests from users, e.g., usersof the backup server 16 utilizing the disk appliance software 24B shownin FIG. 2, to add additional virtual devices that do not havecorresponding physical devices in attached PTL's. For example, a usermay request to add a second virtual drive 37, when the attached PTL 20has only one physical drive 28. The disk appliance will then emulate therequested additional virtual devices at step 109.

As discussed above in relation to the disk appliance software 24B asshown in FIG. 2, the configuration management task 105 may also modifydata retention policies, replication time periods and other usersettable options.

FIGS. 6 and 7 illustrate certain steps in a process of handling exportof a physical tape from the PTL 20. For example, the backup software 24Amay send a command to the appliance 18 to export from the tape librarythe tape that is in slot 4, designated D in FIG. 7. In other cases, auser with access to the PTL 20 may use a keypad or separate controlinput to the PTL 20 to command export of tape D. In the first case, theappliance 18 receives the command from the backup server. In the secondcase, the appliance 18 receives a message from the PTL 20 that theselected tape is being exported. Thus, tape export requests to the PTLare monitored by the appliance 18 at step 150. The process thencontinues to decision block 155, where a check is made as to whether anyreplication is needed to synchronize the physical tape to be exportedwith the corresponding virtual tape. If the tapes are fully synchronizedthen permission to export the physical tape is issued at step 160. Ifthe physical tape and virtual tape do not match, then replication isperformed at step 165. After the data on the virtual tape is replicatedon the physical tape, then the permission to export the physical tape isissued at step 160. As shown in FIG. 7, when the tape is exported formthe PTL, the corresponding virtual tape is removed from the VTL 22, andthe virtual slot appears empty.

FIG. 8 shows a flow diagram illustrating certain steps in anotherprocess of handling export of a physical tape where a data retentionpolicy is implemented. At decision block 165, it is determined whether atape is being exported. Next, instead of having the virtual tape removedfrom the virtual library 22 upon physical tape export as shown in FIG.7, at step 175 the virtual tape cartridge remains in the virtual librarybut is indicated as write protected, illustrated in FIG. 9A as shaded.In this way, the content of the virtual tape cartridge D will remainsynchronized with exported physical tape cartridge D, but the backupserver will have read access to the files stored there, even though thephysical tape is no longer present, perhaps having been moved to anoffsite remote location for disaster recovery safekeeping. This featureof appliance 18 allows the benefits of both on-site read access toarchived files for fast restores, and simultaneous offsite data storageof the same files in an easily managed way. In advantageous embodiments,a data retention time limit may be selected by the system administratorvia the disk appliance software 24B shown in FIG. 2.

While a virtual tape is write-protected, the virtual slot that it isstored in (virtual slot V4 in this example) is not available forimportation of a new virtual tape. If a new physical tape D′ (e.g., areplacement tape for the exported physical tape) is imported to the PTL20 in the physical slot 4, then it will be detected at step 185, alsoillustrated in FIG. 9B. A check is made at decision block 190 as towhether or not the write protection of the virtual tape VD located inthe virtual slot V4 where the new physical tape was inserted hasexpired. If the write protection time has been reached, then the writeprotection is canceled at step 195 and a new virtual tape VD′ isemulated (step 200) in the virtual slot V4 where the expired writeprotected virtual tape D was located as shown in FIG. 9C. If the writeprotection has not expired, then the disk appliance masks the importedtape at step 205. During backups, the backup software will utilize spaceon the non-write protected tapes until the write protection for the tapecartridge VD expires and the cartridge VD is exported from the VTL 22.

FIG. 10 shows a flow diagram illustrating certain steps in an exemplaryprocess of utilizing the virtual tape shelf (VTS) 26 as discussed aboveand shown in FIG. 2. The exporting of physical tape D from PTL 20 willbe used as an example embodiment of a way to use the VTS 26 and theprocess of FIG. 10. Configurations of PTL 20, VTL 22, and VTS 26 areillustrated in FIGS. 11A and 11B. Starting at step 210, a data storagearea is allocated to the VTS 26. Step 210 may be performed as part ofthe configuration management step 105 shown in FIG. 4A. The VTS 26 mayinclude one or more virtual tape drives 52, 54, as well as a virtualrobotic cartridge exchanger 56. Also included are multiple virtual tapestorage slots, designated V1 through V6 in FIGS. 11A and 11B. At step215, a command to the PTL 20 corresponding to removal or export ofphysical tape D is detected. At step 220, in response to the detectedremoval of physical tape D, the virtual tape VD disappears from VTL 22and appears in a storage slot of VTS 26. The virtual tape D is stillavailable to the disk appliance 18 when it is in the VTS 26.

While the above detailed description has shown, described, and pointedout novel features of the invention as applied to various embodiments,it will be understood that various omissions, substitutions, and changesin the form and details of the device or process illustrated may be madeby those skilled in the art without departing from the spirit of theinvention. As will be recognized, the present invention may be embodiedwithin a form that does not provide all of the features and benefits setforth herein, as some features may be used or practiced separately fromothers.

What is claimed is:
 1. A method of emulating a tape library data storage system using one or more hard disk drives, the method comprising: querying one or more physical tape libraries in a data storage system to acquire a configuration of the one or more physical tape libraries in the data storage system; allocating data storage space on the one or more hard disk drives to virtual devices of one or more virtual tape libraries, wherein the one or more virtual tape libraries comprise virtual devices emulating physical devices in the acquired configuration of the one or more physical tape libraries in the tape storage system; and allocating data storage space on the one or more hard disk drives to at least one additional virtual device associated with the one or more virtual tape libraries, wherein the additional virtual device has no corresponding physical device in the data storage system.
 2. The method of claim 1, wherein said additional virtual device comprises a virtual tape drive.
 3. The method of claim 1, wherein said additional virtual device comprises a tape cartridge storage slot.
 4. A method of emulating data storage on a magnetic tape media using one or more hard disk drives, the method comprising: allocating data storage space of the one or more hard disk drives to one or more virtual tape libraries, wherein the one or more virtual tape libraries comprise one or more virtual devices emulating states of one or more physical devices of the one or more physical tape libraries in a tape storage system; storing data in the storage space allocated to the one or more virtual tape libraries according to a first user defined periodic schedule; and replicating the data stored on the one or more virtual tape libraries onto the one or more physical tape libraries according to a second user defined periodic schedule.
 5. The method of claim 4, further comprising: detecting a request to export one of one or more physical tapes from one of the one or more physical tape libraries; and replicating data from a virtual tape corresponding to the physical tape requested to be exported to the physical tape requested to be exported before the physical tape is exported, so as to synchronize the physical tape with the virtual tape.
 6. A method of handling data storage on a hard disk storage system implemented to emulate one or more attached tape libraries, said method comprising: requesting export of at least one physical tape from a physical tape library; and write-protecting the data on the hard disk storage system that is associated with the virtual tape corresponding to the physical tape to be exported.
 7. The method of claim 6, additionally comprising exporting the physical tape.
 8. The method of claim 7, additionally comprising reading said write protected data from said virtual tape after said physical tape is exported.
 9. The method of claim 6, additionally comprising setting a time limit for expiration of said write-protection.
 10. A method of emulating a tape library data storage system using one or more hard disk drives, the method comprising: allocating data storage space on the one or more hard disk drives to virtual devices of one or more virtual tape libraries, wherein the one or more virtual tape libraries comprise virtual devices emulating physical devices of one or more physical tape libraries in the tape storage system; and allocating data storage space on the one or more hard disk drives to at least one additional virtual tape library that has no corresponding physical tape library in the data storage system.
 11. The method of claim 11, additionally comprising moving virtual tapes from a virtual library that corresponds to a physical library to said additional virtual tape library.
 12. The method of claim 11, wherein said moving is performed in response to a command to export a physical tape from a physical tape library.
 13. A storage system comprising at least one disk based storage appliance and at least one tape library, wherein the disk based storage appliance is configured to respond to commands generated by backup software as at least one emulated tape library, and wherein said disk based storage appliance stores data files in an emulated tape library accessible to said backup software that are also stored on tapes that have been previously removed from said tape library, whereby tape archive and disk based read access to a set of data files is simultaneously provided.
 14. The storage system of claim 12, wherein said data files are write protected in said disk based appliance. 